UNCLASSIFIED 


Defense  Technical  Information  Center 
Compilation  Part  Notice 

ADP010617 

TITLE:  Methods  for  Improving  Depth  Perception  in 
HMDS 

DISTRIBUTION:  Approved  for  public  release,  distribution  unlimited 


This  paper  is  part  of  the  following  report: 

TITLE:  The  Capability  of  Virtual  Reality  to  Meet 
Military  Requirements  [la  Capacite  de  la  realite 
virtuelle  a repondre  aux  besoins  militaires] 

To  order  the  complete  compilation  report,  use:  ADA388966 

The  component  part  is  provided  here  to  allow  users  access  to  individually  authored  sections 
of  proceedings,  annals,  symposia,  ect.  However,  the  component  should  be  considered  within 
the  context  of  the  overall  compilation  report  and  not  as  a stand-alone  technical  report. 

The  following  component  part  numbers  comprise  the  compilation  report: 

ADPO 10609  thru  ADPO 10633 


UNCLASSIFIED 


9-1 


METHODS  FOR  IMPROVING  DEPTH  PERCEPTION  IN  HMDS 

Robert  E.  Fischer3,  Daniel  J.  Reiley3,  Charles  Popeb,  and  Eli  Pelic 

a.  OPTICS  1,  Inc.,  Westlake  Village,  CA  91359,  USA 

b.  NAWC  Training  Systems  Division,  Orlando,  FL  32826,  USA 

c.  The  Schepens  Eye  Research  Institute,  Harvard  Medical  School,  Boston,  MA,  USA 

ABSTRACT 

Head  mounted  display  systems  typically  present  imagery  to  the  user  focused  either  at  infinity  (collimated  light  into  the 
eyes),  or  alternatively  at  some  nominal  finite  distance  in  the  order  of  11  - 18  feet  (diverging  light  into  the  eyes).  When 
the  imagery  presented  by  an  HMD  is  focused  at  a finite  distance,  the  right  and  left  eyes  are  sometimes  intentionally  set  to 
converge  to  that  same  distance.  Alternatively,  the  eyes  are  often  left  viewing  parallel  to  one  another.  In  some  HMDs  the 
user  is  permitted  to  select  his  or  her  own  preferred  focus  distance.  There  appears  to  be  an  advantage  in  improving  depth 
perception  in  an  HMD  simulation  environment  by  altering  in  real  time  the  focus  or  apparent  object  distance  to  match  the 
distance  of  the  principal  object  or  objects  being  viewed  at  that  time  by  the  user.  An  eye  tracker  may  be  employed  to 
determine  where  in  the  scene  the  user  is  looking,  and  the  data  is  fed  back  to  the  computer  to  perform,  as  appropriate,  the 
refocusing  task  based  on  the  known  distance  of  the  object  being  looked  at.  Further  realism  improvement  is  likely  by 
changing  the  convergence  as  well.  This  paper  will  discuss  the  relative  merits  of  altering  the  accommodation  and 
convergence  as  well  as  the  means  for  accomplishing  the  refocusing  task  in  the  HMD  viewing  optics  rapidly,  in  real  time, 
and  without  otherwise  altering  the  image  quality  or  magnification.  The  net  goal  is  to  improve  the  overall  realism  of  the 
simulation  to  the  user. 

INTRODUCTION 

Head-mounted  displays  (HMDs)  have  found  widespread  use  in  a variety  of  applications.  Their  application  for  pilot 
training  has  been  especially  successful  because  they  are  much  less  expensive  and  less  dangerous  than  actual  flying,  and 
far  less  expensive  to  implement  than  large  dome  simulators. 

To  achieve  the  best  results  in  training,  the  training  environment  presented  by  the  HMD  should  be  as  close  as  possible  to 
the  actual  environment  experienced  by  the  pilot.  One  of  the  most  important  improvements  in  HMDs  is  the  accurate 
simulation  of  depth  and  three-dimensional  images. 

The  need  for  high  quality  three-dimensional  imaging  in  flight  simulation  and  other  applications  has  driven  the 
development  of  various  technologies  (Okoshi,  1980)).  Many  of  these  technologies  are  fairly  complex  (i.e.  Williams, 
1988)  and  are  still  far  from  implementation.  As  a result  the  only  technique  currently  being  used  in  such  applications  is 
dichoptic  (separate  image  to  each  eye)  presentation  of  binocular  disparity.  This  paper  describes  preliminary  efforts  at 
OPTICS  1 to  improve  the  realism  of  HMDs  and  to  determine  the  merits  and  viability  of  adjusting  in  real  time  the 
accommodation  or  apparent  object  distance  as  well  as  the  convergence  in  an  HMD. 

SPECIFICATIONS  AND  REQUIREMENTS 

The  HMD  specifications  to  which  we  will  work  are  summarized  in  Table  1 below.  These  data  represent  our  best  effort  at 
describing  a viable  and  achievable  HMD  implementation  with  good  performance. 


Paper  presented  at  the  RTO  HFM  Workshop  on  "The  Capability  of  Virtual  Reality  to  Meet  Military 
Requirements” , held  in  Orlando,  USA,  5-9  December  1997,  and  published  in  RTO  MP-54. 
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Parameter 

Specification 

Full  diagonal  field  of  view 

40  degrees 

Exit  pupil  diameter 

7mm  minimum,  10mm  goal 

Eye  relief 

25.4mm  minimum 

Exit  pupil  location 

Always  at  eye  pupils 

Interpupillary  distance,  nominal 

62  mm 

Range  for  mterpup illary  distance 

52  - 72  mm 

Distortion 

3%  maximum 

Image  blur  within  horizontal  inscribed  circle 

2.5  minutes  of  arc  maximum 

Image  blur  to  comer 

3.5  minutes  of  arc 

Spectral  band 

Visual 

Spectral  weighting 

Photopic  eye  response 

Minimum  accommodation  distance 

15  inches 

Maximum  accommodation  distance 

Infinity 

Convergence  at  infinity 

Zero 

Convergence  at  15  inches 

9.3  degrees 

Speed  of  change  in  accommodation 

<0.6  seconds 

Magnification  change  over  accommodation  range 

4%  maximum 

Eye  tracker 

Yes 

Table  1. 


VISUAL  EFFECTS  AND  MERITS  OF  VARYING  ACCOMMODATION  AND  CONVERGENCE  TO 
ENHANCE  REALISM 

Depth  Perception  in  a Natural  Environment 

The  human  mind  uses  a wide  variety  of  clues  to  determine  distance  and  depth.  These  clues  include  angular  extent  of  an 
object,  occlusion  by  other  objects,  perspective,  convergence,  and  accommodation.  Each  of  these  clues  need  to  be 
carefully  considered  in  the  complete  implementation  of  an  HMD  system.  However,  only  convergence  and 
accommodation  can  be  addressed  in  the  physical  and  optical  design  of  an  HMD  system,  so  these  will  be  the  only  topics 
addressed  in  this  paper. 

Convergence  is  the  rotation  of  the  eyes  towards  an  object.  As  a person  looks  at  an  object,  the  eyes  rotate  so  that  the 
object  is  imaged  onto  the  fovea  of  the  retina  which  is  the  small  region  of  highest  visual  acuity.  This  rotation  means  that 
the  centerlines  of  the  eyes  intersect  at  the  object.  Convergence  is  measured  by  the  total  included  angle  between  the  eye’s 
centerlines.  This  angle  between  the  eyes  is  one  of  the  most  important  visual  clues  to  depth.  Unfortunately  for  HMD 
designers,  this  angle  is  a function  of  the  user’s  interpupillary  distance  or  IPD. 

Accommodation  is  the  refocusing  of  the  eyes  to  the  distance  of  the  object  being  viewed. 

In  a natural  viewing  environment,  accommodation  and  convergence  work  together  to  aid  in  determining  distance.  The 
distance  at  which  the  eyes  are  focussed  is  the  same  as  the  distance  at  which  the  eyes’  centerlines  intersect. 

Depth  Perception  in  an  Artificial  Environment 

HMD  designers  must  try  to  approximate  the  natural  environment  depth  perception  as  well  as  possible.  However,  they  are 
subject  to  several  constraints  which  should  also  guide  their  design  considerations.  First,  because  of  the  inter-relationship 
between  the  IPD  and  the  convergence  angle,  an  exact  match  between  accommodation  distance  and  convergence  angle  is 
possible  for  only  a small  set  of  cases.  Second,  because  objects  of  varying  distances  may  be  represented  on  the  same 
viewing  screen,  some  mismatch  between  accommodation  distance  and  represented  distance  is  inevitable. 

Two  measures  describe  the  degree  to  which  an  HMD  differs  from  the  natural  viewing  environment.  The  first  measure  is 
called  “accommodation  error”  and  is  a measure  of  the  difference  between  the  distance  at  which  the  eye  is  focussed  and 
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the  distance  at  which  the  object  is  represented.  The  second  measure  is  called  “convergence  error”  and  is  a measure  of  the 
difference  between  the  convergence  angle  required  by  the  displayed  objects  distance  and  the  convergence  angle 
presented  by  the  HMD. 

Many  difficulties  occur  when  the  convergence  error  and  accommodation  error  are  too  large.  These  difficulties  include 
loss  of  realism,  headaches,  double  vision,  blurred  vision,  and,  in  extreme  instances,  even  nausea.  For  small  errors,  these 
difficulties  tend  to  ease  as  the  duration  of  HMD  use  increases.  For  larger  errors  and  longer  duration  of  HMD  use, 
however,  these  difficulties  can  persist  even  after  the  used  of  the  HMD. 

Determining  the  acceptable  errors  in  accommodation  and  convergence  angle  is  an  active  area  of  research.  The  HMD 
design  community  should  probably  never  expect  to  obtain  definitive,  universally  applicable  results  for  the  acceptable 
errors  for  several  reasons.  First,  this  research  is,  by  definition,  very  subjective  and  dependent  on  the  particular 
experimental  parameters.  Second,  different  situations  will  have  different  acceptable  errors;  the  requirements  on  a HMD 
for  a military  flight  simulator  are  very  likely  to  be  different  from  those  of  an  inexpensive  video  game.  Third,  everyone’s 
visual  system  is  different;  a wide  variety  of  common  eye  conditions  can  cause  people  to  be  unusually  sensitive  or 
unusually  lax  in  their  tolerance  to  convergence  error  or  accommodation  error.  Finally,  people  have  a wide  variety  of 
tolerances  to  non-ideal  situations,  due  merely  to  their  personalities,  moods,  and  temperaments. 

To  precisely  investigate  visual  effects  and  merits  of  varying  accommodation  and  convergence,  it  is  important  to  precisely 
define  our  terms.  Accommodation  error  is 

A - 1 /s„  -1  /Sf, 

where  s0  is  the  object  distance  to  be  represented,  st  is  the  distance  at  which  the  HMD  is  focussed,  and  A is  the 
accommodation  error.  Accomodation  error  is  usually  expressed  in  diopters.  (1  diopter  = 1/meter)  The  ideal 
convergence  angle  is 

1 A ,//  II  L>I, So, 

where  1PD  is  the  interpupillary  distance,  and  CAideal  is  the  ideal  convergence  angle.  The  convergence  angle  error  is 

D CA  — CAreal  - CAjdeal, 

where  Oca  is  the  convergence  angle  error,  and  CAreai  is  the  convergence  angle  presented  to  the  user.  Convergence  error 
can  be  expressed  in  cither  degrees  or  prism  diopters.  (1  prism  diopter  = 0.01  radian  = 0.57  degrees). 

Nevertheless,  it  is  helpful  to  investigate  the  requirements  of  an  HMD  for  various  assumptions  of  convergence  angle  error 
and  accommodation  error. 

Figure  1 shows  how  the  convergence  and  accommodation  interrelate,  and  how  errors  in  both  can  happen. 
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Figure  1 - Simulating  Distance  in  a Real  HMD 

• Exact  accommodation  and  convergence  can  only  be  obtained 
for  a small  set  of  circumstances 

• Accommodation  error:  eyepieces  focussed  at  wrong  distance 

• Convergence  error:  boresights  intersect  at  wrong  distance 


error 


The  top  line  in  the  figure  shows  that,  for  an  acceptable  accommodation  error  of  1/16  diopter,  16  focus  positions  would  be 
required  to  represent  the  range  of  object  positions;  for  object  positions  between  0.5m  and  2m,  a near-continuum  of 
positions  is  required.  The  bottom  line  on  the  figure  shows  that,  for  an  acceptable  accommodation  error  of  1/3  diopter, 
only  three  focus  positions  are  required.  Figure  3 shows  the  required  convergence  angles  for  object  distances  from  0.5 
meter  and  above.  The  two  curves  are  for  convergence  errors  of  1 .5  degrees  and  0.2  degrees. 

Figure  2 shows  the  required  focus  positions  for  object  distances  from  0.5m  to  infinity  for  two  acceptable 
accommodation  errors. 


Figure  2 - Accommodation  and  HMD  Design 

° “ INCORPORATED 


• Focus  positions  to  represent  object  distances  from  0.5m  to  infinity: 

• for  1/2  diopter  accommodation  error,  only  two  focus  positions 
are  required 

• for  1/16  diopter  accommodation  error,  16  positions  are  required 
(continuous  focus  might  be  better) 
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Figure  3 - Convergence  in  HMD  Design 


OPTICS  1 


Convergence 


Object  distance,  meters 


* Convergence  angles  to  represent  object  distances  from  0.5m  to  infinity 

• for  1.5  degree  convergence  error,  six  convergence  angles  are  required 

* for  0.2  degree  convergence  error,  12  convergence  angles  are  required 
(continuous  adjustment  might  be  better) 


Accommodation  Methods 

A wide  variety  of  accommodation  methods  are  feasible.  These  methods  include,  in  no  particular  order: 

1.  Translating  the  lens  assembly 

2.  Translating  the  display  device 

3.  Translating  a sub-group  of  lens  elements 

4.  The  Alvarez  asphere,  bifocal/multifocal  lenses 

5.  Bifocal  or  multifocal  lenses 

6.  Liquid  filled  lens 

7.  Liquid  crystal  lens 

8.  Adaptive  optics 

1.  Translating  the  lens  assembly  relative  to  the  display  device  changes  the  accommodation  distance  from  infinity 
to  a closer  image  distance.  Presenting  images  at  infinity  and  at  0.5m  requires  moving  the  lens  assembly  about  3mm 
for  a 35mm  focal  length  lens.  This  method  of  accommodation  has  several  advantages,  most  importantly,  it  is  easy  to 
implement.  This  method  has  some  disadvantages,  too:  it  requires  moving  parts.  A potential  disadvantage  is  that  the 
exit  pupil  location  where  the  eye  is  located  may  move  causing  vignetting  or  clipping  of  light  at  off  axis  field 
positions. 

2.  Translating  display  device  switches  from  a distant  image  distance  to  a close  image  distance  by  moving  the 
display  device  from  the  back  focus  of  the  lens  to  a position  inside  the  back  focus.  This  is  similar  to  translating  the 
lens  assembly.  It  has  the  advantage  of  not  changing  the  stop  position  if  the  viewing  optics  is  designed  to  be 
“telecentric”  at  the  display.  Unfortunately,  it  has  a serious  disadvantage  in  that  display  mounts  are  usually  fairly 
complex  and  would  be  difficult  to  move. 

3.  Translating  a sub-group  of  lens  elements  within  the  lens  assembly  switches  from  a distant  image  distance  to  a 
close  image  distance  by  changing  the  lens  assembly’s  focal  length  much  as  in  a zoom  lens.  This  method  has  the 
advantage  that  it  can  be  easy  to  implement  and  that  the  moving  mass  can  be  very  small.  Its  disadvantages  include 
more  complex  lens  housing  and  the  inherent  disadvantages  of  moving  parts. 


9-6 


4.  The  Alvarez  asphere  switches  from  a distant  image  distance  to  a close  image  distance  by  moving  special 
aspheric  surfaces  perpendicular  to  the  optical  axis.  A similar  method  was  used  in  the  autofocus  mechanism  in  the 
Polaroid  Spectra  camera.  This  method  has  several  advantages:  only  a small  mass  must  be  moved,  the  center  of  mass 
does  not  change,  and  only  a small  motion  is  required.  Further,  this  motion  is  linear.  This  method  also  has  several 
disadvantages:  the  aspheric  surfaces  are  difficult  to  fabricate  because  they  are  not  rotationally  symmetric,  and  the 
effect  on  image  quality  can  be  unacceptable. 

5.  A bifocal  or  multifocal  lens  can  be  constructed  such  than  some  regions  of  the  lens  form  a distant  image  and 
other  regions  form  a close  image.  This  method  is  particularly  attractive  for  applications  which  only  require  two 
focus  positions,  and  because  no  moving  parts  are  required.  Disadvantages  of  this  method  include  lowered  resolution 
and  fabrication  difficulties. 

6.  A liquid  filled  lens  has  been  used  in  some  applications  requiring  a variety  of  focus  positions.  The  liquid  is  held 
between  compliant  thm  windows  that  form  the  outer  optical  surfaces.  As  liquid  is  pumped  into  or  out  of  the  lens,  the 
compliant  walls  change  shape,  causing  a change  in  focal  length  of  the  device.  This  method  has  the  advantages  of 
having  no  moving  optical  parts  and  of  the  potential  for  quick  adjustment  times.  This  method  has  the  disadvantages 
of  a limited  accommodation  and  all  the  troubles  that  come  with  the  use  of  a liquid. 

7.  Liquid  crystal  lenses  have  been  used  in  many  applications  requiring  a quick  adjustment  in  focal  length.  In  a 
liquid  crystal  lens,  the  liquid  crystal  is  placed  inside  of  a lens-shaped  cavity.  When  an  electric  field  is  applied  to  a 
liquid  crystal,  the  molecules  tend  to  align.  Their  uniaxial  or  biaxial  symmetry  creates  a phase  retarder.  The  liquid 
crystal  lens  is  made  without  polarizers.  Rather  than  create  a quadratic  phase  variation,  researchers  have  made  liquid 
crystal  Fresnel  lenses.  As  voltage  is  applied  to  the  liquid  crystal,  its  index  changes,  causing  a change  in  the  focal 
length  of  the  lens.  This  method  has  the  advantages  of  having  no  moving  parts,  the  potential  for  very  fast  adjustment 
speed,  and  maintaining  the  correct  pupil  position.  This  method  has  the  disadvantage  of  having  a limited 
accommodation,  as  well  as  having  all  the  disadvantages  of  using  a liquid  crystal. 

8.  Adaptive  optics  are  also  often  used  in  changing  the  phase  profile  of  light.  Adaptive  optics  are  lenses  or  mirrors 
which  change  shape  using  a variety  of  actuator  methods.  This  method  has  the  advantage  that  a continuous 
adaptation  can  be  provided.  However,  adaptive  optics  usually  suffer  from  the  disadvantages  of  being  difficult  to 
control;  furthermore,  they  are  a new  technology  and  therefore  tend  to  be  expensive. 

CONVERGENCE  METHODS 

A wide  variety  of  convergence  methods  are  also  available.  These  methods  include,  in  no  particular  order,  the  following: 

1.  Video  centering 

2.  Translating  the  display 

3.  Tilting  the  entire  assembly 

4.  Optical  wedges  in  front  of  the 
eyes 

1.  Video  centering  achieves  convergence  by  electronically  moving  the  center  of  the  scene  away  from  the  center  of 
the  display  device.  For  example,  when  simulating  the  convergence  of  an  object  at  infinity,  the  images  will  be  moved 
towards  the  nose  to  create  Ihe  desired  convergence  angle.  This  method  has  the  advantage  of  having  no  moving  parts 
and  allowing  quick  convergence  times.  Its  disadvantages  include  more  complex  electronics,  rotation  about  the  pupil, 
and  the  fact  that  the  display  must  in  effect  be  larger  than  what  is  required  for  any  given  convergence  situation. 
Figure  4 shows  a graphical  representation  of  the  video  centering  methodology. 
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Figure  4 - Video  Centering  for  Convergence 
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• Advantages 

• no  moving  parts 

• simpler  mechanics 


Disadvantages 

• more  complex  electronics 

• rotation  about  pupil 

• Require  oversize  display 


Figure  4 


2.  Convergence  can  also  be  achieved  by  translating  the  entire  display  devices  towards  the  nose.  This  is  often  a 
problem  as  the  electro-mechanical  implementation  may  be  difficult. 

3.  The  entire  viewing  optics  assembly  including  the  optics  and  the  display  can  be  tilted  to  match  the  desired 
convergence  angle.  Ideally  this  tilt  angle  should  be  at  about  the  eye  pupil  location  so  as  to  avoid  vignetting  or  other 
bad  optical  effects.  While  this  is  a difficult  implementation  task,  it  correlates  directly  with  the  desired  result,  making 
it  somewhat  attractive. 

4.  We  can  locate  optical  wedges  in  front  of  the  eye  so  as  to  vary  the  magnitude  of  convergence  presented  to  each 
eye.  If  two  optical  wedges  or  weak  prisms  are  counter-rotated  we  can  achieve  a variable  degree  of  convergence.  The 
disadvantages  include  the  mechanization  complexity  as  well  as  lateral  color  or  color  fringing  around  objects.  A 
liquid  filled  variable  wedge  is  also  feasible. 

EXAMPLE  OF  A CANDIDATE  IMPLEMENTATION 

In  order  to  give  a realistic  example  of  a viable  implementation,  we  have  modified  our  PT-Ol  HMD  product  in  order  to 
allow  for  a rapid  change  in  both  accommodation  and  convergence  from  infinity  to  a distance  of  approximately  15  inches. 
This  HMD  is  a commercial  product  used  in  video  inspection,  cinematography,  and  other  professional  areas.  Its  field  of 
view  is  27.5  degrees  diagonal  and  thus  not  ideal  for  a simulation  application,  however  we  used  it  here  due  to  its 
availability. 

The  convergence  change  was  accomplished  by  rotating  the  viewing  optics  and  display  assemblies  in  front  of  each  eye  by 
approximately  4.6  degrees  towards  the  nose,  and  the  accommodation  was  accomplished  by  refocusing  each  of  the 
viewing  optics  lens  assemblies  by  3.8  mm  towards  the  LCD  display  device.  As  it  was  not  possible  to  rotate  the 
assemblies  about  the  ideal  eye  pupil  location,  the  performance  is  not  ideal.  However,  it  does  demonstrate  the  feasibility 
and  the  overall  performance  of  such  an  implementation. 
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Figure  5 shows  the  lens  design  for  the  HMD  as  well  as  its  optical  performance  in  the  form  of  geometrical  spot  diagrams. 
We  show  that  the  imagery  at  the  very  corner  of  the  field  of  view  is  in  the  order  of  5 arc  minutes  rms  blur  diameter  which 
is  reasonable  for  this  application.  We  also  show  the  lens  assembly  and  display  device  rotated  towards  the  nose  by  4.6 
degrees. 
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SUMMARY  AND  CONCLUSIONS 

The  literature  clearly  shows  the  importance  of  providing  the  cues  of  accommodation  and  convergence  in  an  HMD  when 
used  in  a simulation  environment.  This  is  especially  true  when  the  user  is  in  a simulated  cockpit  or  similar  environment 
where  imagery  of  close  objects  must  be  simulated  as  well  as  imagery  of  objects  at  infinity  and/or  other  distances. 

We  have  shown  the  accommodation  and  convergence  demands  for  a typical  HMD  as  used  in  a simulation  environment. 
It  is  important  that  the  accommodation  and  convergence  demands  both  be  appropriately  coupled.  Eight  different 
methods  for  implementation  of  a variable  accommodation  and  4 methods  for  variable  convergence  in  an  HMD  were 
presented.  The  different  methods  differ  from  one  another  in  complexity,  performance,  and  other  factors.  Additional 
work  will  be  directed  towards  identifying  the  optimum  system  for  implementation. 
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